Method and apparatus for summarization of unsupervised video with efficient key frame selection reward functions

ABSTRACT

Disclosed are a method and apparatus for summarization of unsupervised video with efficient key frame selection reward functions. Frame-level visual features are extracted from an input video. An attention weight is computed and an importance score is represented as a frame tracking probability for selecting a key frame using the attention weight. A temporal consistency reward function and a representativeness reward function are obtained so as to select the key frame, based on a visual similarity distance and temporal distance between key frames, and an attention-based video summarization network is trained to predict an importance score for selecting a key frame of a video summary by using the temporal consistency reward function and the representativeness reward function. A video summary is created by selecting a corresponding key frame based on the predicted importance score, the quality of the created video summary is evaluated, and policy gradient learning is performed for the attention-based video summarization network. Regularization and reconstruction loss is calculated for controlling the probability to select a key frame by using the importance score of the selected key frame. A video summary is created based on the calculated regularization and reconstruction loss.

CROSS-REFERENCES TO RELATED APPLICATION

This application is based on and claims priority under 35 U.S.C. 119 toKorean Patent Application No. 10-2022-0026671, filed on Feb. 03, 2022 inthe Korean intellectual property office, the disclosures of which areherein incorporated by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates to a method and apparatus forsummarization of unsupervised video with efficient key frame selectionreward functions.

BACKGROUND OF THE INVENTION

Many people spend their time browsing videos that interest them inonline video sharing platforms like YouTube® (YouTube is a Trademark ofGoogle, LLC, in the U.S. and elsewhere). As a way to save them timebrowsing, people usually use a preview or a short video summary to getan efficient and quick grasp of the entire video content [1]. Sincevideo summarization has been important over the past few years, activeresearch is being conducted to browse video content and/or to make ashort summary video out of a long video. One problem with the task ofvideo summarization is that can be challenging to predict frame-level orshot-level importance scores of videos [2]. This is because videosummarization is generally an abstract and subjective multi-mode taskthat has no explicit audiovisual pattern or semantic rules. If one findsa frame of a video interesting and useful, the importance score of thatframe should usually be high. Such frames with high scores can beselected to create a video summary.

Various methods have been proposed recently, and these methods show highperformance using deep learning [3], [4], and [5]. Deep learning-basedvideo summarization methods are divided into supervised learning-basedmethods and unsupervised learning-based methods. In the case of thesupervised learning-based methods, it is often quite difficult to createa labeled dataset. It is also generally quite difficult to create alarge dataset that encompasses various domains or scenes. For thisreason, more focus has recently been put on the development ofunsupervised video summarization methods.

A reinforcement learning (RL)-based video summarization method wasproposed in the conventional technology [6] and has shown good results.Notably, there is an efficient and explicit evaluation method forselecting a key frame, which is a reward function, to train a deepneural network using RL. Also, the deep neural network efficientlylearns various features of videos, such as representativeness,diversity, and integrity, using an evaluation method. Using RL, theconventional technology [3] uses piecewise linear interpolation proposedIntern-SUM. The use of interpolation reduced network output helps toalleviate the high distribution problem and improves performance.However, in some, case key frames were selected only from a particularscene in many videos, or interesting key frames were seldom selected.

Moreover, there are a few drawbacks with the existing RL-based videosummarization methods. First, it is often difficult to capture visualand temporal context with a deep neural network. Second, many of thesemethods do not allow for temporal distribution of key frames but use areward function or loss function in order to train a network bycalculating visual differences between key frames. Therefore, there is aneed for a video summarization method that conveys a producer’sstoryline by selecting a key frame in a way that makes it easy tounderstand a video.

SUMMARY

An aspect of the present disclosure proposes a reinforcementlearning-based video summarization framework with new temporalconsistency reward and representativeness reward functions (TR-SUM) andan attention-based video summarization method and apparatus forprecisely predicting importance scores. More specifically, an aspect ofthe present disclosure provides a video summarization network having anattention-based encoder-decoder architecture that efficiently capturesthe context of a long video and new reward functions, which are atemporal consistency reward function and a representativeness rewardfunction, for efficiently and uniformly selecting a key frame ofinterest.

In one aspect, an unsupervised video summarization method with anefficient key frame selection reward function according to the presentdisclosure includes: extracting frame-level visual features from aninput video; computing an attention weight and representing animportance score as a frame tracking probability for selecting a keyframe by using the attention weight; obtaining a temporal consistencyreward function and a representativeness reward function so as to selectthe key frame, based on a visual similarity distance and temporaldistance between key frames, and training an attention-based videosummarization network to predict an importance score for selecting a keyframe of a video summary by using the temporal consistency rewardfunction and the representativeness reward function; creating a videosummary by selecting a corresponding key frame based on the predictedimportance score, evaluating the quality of the created video summary,and performing policy gradient learning for the attention-based videosummarization network; calculating regularization and reconstructionloss for controlling the probability to select a key frame by using theimportance score of the selected key frame; and creating a video summarybased on the calculated regularization and reconstruction loss.

In the step of computing the attention weight and representing theimportance score as a frame tracking probability for selecting a keyframe by using the attention weight, the step being performed using anencoder network, a decoder network, and an attention layer between theencoder network and the decoder network reduce parameters andcalculations through dilated RNN and extract temporal dependency,wherein the encoder network captures visual similarities with local andglobal context between key frames, wherein the attention layer computesan attention weight by using both the output of the encoder network andthe last hidden state of the decoder network, wherein the attentionweight is normalized to a probability score of each key frame by asoftmax function, wherein a context vector is obtained by multiplyingthe output of the encoder network by the attention weight, and whereinthe decoder network is trained by connecting the context vector and theprevious output of an initialized decoder network for the input of thedecoder network, to obtain an importance score by using learning resultsof the decoder network and the encoder network.

In the step of obtaining the temporal consistency reward function andthe representativeness reward function so as to select the key frame,based on a visual similarity distance and temporal distance between keyframes, and trains an attention-based video summarization network topredict an importance score for selecting a key frame of a video summaryby using the temporal consistency reward function and therepresentativeness reward function, the step further includescalculating the similarity between the selected key frame and all keyframes of the video is calculated by using features extracted using therepresentative reward function and predicting an importance score forselecting a key frame of a video summary through the representativereward function, and repeating the process of finding the closestneighbor among the selected key frames with respect to all the keyframes, in order to select a representative shot-level key frame usingthe temporal consistency reward function.

In the step of creating a video summary by selecting the correspondingkey frame based on the predicted importance score, evaluating thequality of the created video summary, and performing policy gradientlearning for the attention-based video summarization network, the stepfurther includes performing parameterized policy gradient learning bycomputing a set of normalized importance weights for each episode byusing an objective function of an exploration strategy for exploring anunder-appreciated reward (UREX) method and a softmax function forapproximation of the objective function.

In another aspect, an attention-based video summarization apparatusaccording to the present disclosure includes: a processor; and a memorycoupled to the processor, the memory containing instructions, that whenexecuted by the processor: extracts frame-level visual features from aninput video; computes an attention weight through an attention-basedvideo summarization network and represents an importance score as aframe tracking probability for selecting a key frame by using theattention weight; obtains a temporal consistency reward function and arepresentativeness reward function so as to select the key frame, basedon a visual similarity distance and temporal distance between keyframes, and trains an attention-based video summarization network topredict an importance score for selecting a key frame of a video summaryby using the temporal consistency reward function and therepresentativeness reward function; creates a video summary by selectinga corresponding key frame based on the predicted importance score,evaluates the quality of the created video summary, and performs policygradient learning for the attention-based video summarization network,wherein the video summarization network module calculates regularizationand reconstruction loss for controlling the probability to select a keyframe by using the importance score of the selected key frame; andcreates a video summary based on the calculated regularization andreconstruction loss.

According to embodiments of the present disclosure, there are proposed areinforcement learning-based video summarization framework with newtemporal consistency reward and representativeness reward functions(TR-SUM) and an attention-based video summarization method and apparatusfor precisely predicting importance scores. The proposed unsupervisedvideo summarization method may efficiently capture the context of a longvideo through a video summarization network having an attention-basedencoder-decoder architecture and may efficiently and uniformly select akey frame of interest using new reward functions which are a temporalconsistency reward function and a representativeness reward function.

DESCRIPTION OF THE DRAWINGS

The foregoing aspects and many of the attendant advantages of thisinvention will become more readily appreciated as the same become betterunderstood by reference to the following detailed description, whentaken in conjunction with the accompanying drawings, wherein:

FIG. 1 is a view for explaining the concept of an attention-based videosummarization network according to an embodiment of the presentdisclosure.

FIG. 2 is a view showing a reinforcement learning-based videosummarization framework using temporal consistency reward andrepresentativeness reward functions according to an embodiment of thepresent disclosure.

FIG. 3 is a view showing a construction of an unsupervised videosummarization apparatus with an efficient key frame selection rewardfunction according to an embodiment of the present disclosure.

FIG. 4 is a flowchart for explaining an unsupervised video summarymethod with an efficient key frame selection reward function accordingto an embodiment of the present disclosure.

FIG. 5 is a view illustrating a configuration of an attention-basedvideo summarization network for computing an attention weight accordingto an embodiment of the present disclosure.

FIG. 6 is a view for explaining a representativeness reward functionaccording to an embodiment of the present disclosure.

FIG. 7 is a view for explaining a temporal consistency reward functionaccording to an embodiment of the present disclosure.

DETAILED DESCRIPTION

The present disclosure proposes an unsupervised video summarizationmethod with an efficient key frame selection reward function. Areinforcement learning-based video summarization framework with newtemporal consistency reward and representativeness reward functions(TR-SUM) is designed for efficient key frame selection. A videosummarization method and apparatus with an attention-basedencoder-decoder architecture is proposed to predict key frame-levelimportance scores of a video. According to an embodiment of the presentdisclosure, rewards are calculated based on a reward function that helpsefficiently and uniformly select a key frame of interest usingimportance scores.

FIG. 1 is a view 100 for explaining the concept of an attention-basedvideo summarization network according to an embodiment of the presentdisclosure.

The present disclosure proposes a reinforcement learning-based videosummarization framework with new temporal consistency reward andrepresentativeness reward functions (TR-SUM) and proposes anattention-based video summarization network for precisely predictingimportance scores as in FIG. 1 .

The proposed unsupervised video summarization method allows efficientcapturing of the context of a long video through a video summarizationnetwork having an attention-based encoder-decoder architecture, and alsoallows efficient and uniform selection of a key frame of interest usingnovel reward functions which are a temporal consistency reward functionand a representativeness reward function.

The video summarization method is divided into a supervisedlearning-based method and an unsupervised learning-based method. Both ofthe two methods use a video summarization dataset including frame-levelor shot-level importance scores of a video annotated by multiple users[2]. In the supervised learning-based method, a model is trained byusing frame-level or shot-level features of a video as input forpredicting importance scores. In this method, a cost is calculated bythe difference between an importance score predicted using a dataset andan annotated importance score. Also, this method minimizes cost forfinding an optimum model.

The conventional technology [8] proposes a memory augmented videosummarization method. A memory network efficiently provides supportingknowledge extracted from an entire video. To predict the importancescore of a current shot, the score is adjusted by an overallunderstanding of the original video by using a global attentionmechanism. The conventional technology [9] proposes an LSTM-basednetwork with a determinantal point process (DPP) for encoding theprobability to sample a frame for learning representativeness anddiversity. The conventional technology [10] proposes a dilated temporalrelational (DTR) unit of a generator for reinforcing temporal contextrepresentations among video frames. To train a network for predictingthe best summary of a video, an adversarial learning method is usedtogether with a three-player loss function. The conventional technology[11] proposes an attention-based encoder-decoder network to predictimportance scores for key shot selection. This network learns videorepresentations by using an encoder with a bidirectional LSTM networkand a decoder with an attention mechanism.

However, one problem with the supervised learning-based method is thatit is very difficult to make a human-labeled video summarizationdataset, including videos of various domains.

In the conventional technology [4], an attention autoencoder (AAE)network replaces a variational autoencoder (VAE) in SUM-GAN in order toimprove the efficiency and performance of an adversarial autoencoder(AAE) proposed in SUM-GAN. The network gives a weight to an interestingframe for summarizing video during training. The conventional technology[5] proposes a chunk and stride network (CSNet) based on a VAE and GAN(Generative Adversarial Networks) architecture. The conventionaltechnology [12] proposes an adversarial autoencoder (AAE)-based videosummarization model. A selector LSTM selects frames from input framelevel features of a video. Next, the VAE generates a reconstructed videousing the selected frames. A discriminator distinguishes between thereconstructed video and the original input video to train the entirenetwork. For training the model, four different loss functions are used.The conventional technology [13] proposes Cycle-SUM which is a variantof SUM-GAN but adopts a cycle generative adversarial network with twoVAE-based generators and two discriminators in order to preserve theinformation of the original video in the summary video. A videosummarization method proposed in the conventional technology [14] thatuses a Tessellation approach selects a clip that maintains temporalconsistency by finding a visually similar clip and using a Viterbialgorithm which is a graph-based method. The conventional technology[15] proposes an unsupervised learning-based SUM-FCN. This methodproposes a new FCN architecture with temporal convolution converted fromspatial convolution in order to process a video sequence. In thismethod, frames are selected using the output score of the decoder, andthe loss function is calculated with a repelling regularizer to applythe diversity of frames in a summary video.

Deep reinforcement learning combines a deep neural network with areinforcement learning method [16]. A policy gradient method is one ofreinforcement learning methods with no model. The policy gradient methodparameterizes a policy in a deep neural network model and optimizes themodel by maximizing rewards for a state distribution defined by thepolicy using a gradient descent method such as stochastic gradientdescent (SGD). To train the model, this method calculates and minimizescost by an objective function. However, the policy gradient method hasseveral problems, such as the low sample efficiency problem [17] and thehigh distribution problem. Especially in the case of the low sampleefficiency problem, the agent requires many more samples, such as humanexperiences, for learning actions in the environment (states), thanhumans because it is not as intelligent as humans. Another problem isthe high variance of the estimated gradient. This problem is caused bythe high dimensional action space and long-horizon problem [18], whichmeans a hugely delayed reward for a long sequence of decisions to find agoal. In the proposed method, a policy gradient is used along with thebaseline to reduce variance, and the number of episodes is increased toimprove the sample efficiency problem.

FIG. 2 is a view 200 showing a reinforcement learning-based video 102summarization framework using temporal consistency reward andrepresentativeness reward functions 104 according to an embodiment ofthe present disclosure.

In the present disclosure, the video summarization problem is formulatedas a frame selection problem 106 by using importance scores 108predicted by a video summarization network. In particular, a network isdeveloped by using a dilated GRU encoder and a GRU decoder network 110with an attention mechanism 112. This network learns video 102representations and efficiently predicts importance scores 108 byframe-selection probability. The importance scores 108 are convertedinto frame-selection actions 106 to select key frames 114A-N as asummary by using Bernoulli distribution 116, as shown in FIG. 2 .

FIG. 3 is a view showing a construction of an unsupervised videosummarization apparatus with an efficient key frame selection rewardfunction according to an embodiment of the present disclosure.

The proposed unsupervised video summarization apparatus with anefficient key frame selection reward function includes a frame-levelimage feature extraction module 310, a video summarization networkmodule 320, an evaluation module 330, a policy gradient algorithm-basedlearning module 340, and a video summary creation module 350.

The frame-level image feature extraction module 310 according to theembodiment of the present disclosure extracts frame-level visualfeatures from an input video, and the attention-based videosummarization network module 320 computes an attention weight andrepresents an importance score as a frame tracking probability forselecting a key frame by using the attention weight.

In the video summarization network module 320 according to theembodiment of the present disclosure, an encoder network, a decodernetwork, and an attention layer between the encoder network and thedecoder network reduce parameters and calculations through dilated RNNand extract temporal dependency.

In the video summarization network module 320 according to theembodiment of the present disclosure, the encoder network capturesvisual similarities with local and global context between key frames,and the attention layer computes an attention weight by using both theoutput of the encoder network and the last hidden state of the decodernetwork. The video summarization network module 320 according to theembodiment of the present disclosure normalizes the attention weight toa probability score of each key frame by a softmax function and obtainsa context vector by multiplying the output of the encoder network by theattention weight.

The video summarization network module 320 according to the embodimentof the present disclosure trains the decoder network by connecting thecontext vector and the previous output of an initialized decoder networkfor the input of the decoder network, to obtain an importance score byusing learning results of the decoder network and the encoder network.

The evaluation module 330 according to the embodiment of the presentdisclosure obtains a temporal consistency reward function and arepresentativeness reward function so as to select the key frame, basedon a visual similarity distance and temporal distance between keyframes, and trains the attention-based video summarization network topredict an importance score for selecting a key frame of a video summaryby using the temporal consistency reward function and therepresentativeness reward function.

The evaluation module 330 according to the embodiment of the presentdisclosure is trained to calculate the similarity between the selectedkey frame and all key frames of the video by using features extractedusing the representative reward function and predict an importance scorefor selecting a key frame of a video summary through the representativereward function.

The evaluation module 330 according to the embodiment of the presentdisclosure is trained to repeat the process of finding the closestneighbor among the selected key frames with respect to all the keyframes, in order to select a representative shot-level key frame usingthe temporal consistency reward function.

The policy gradient algorithm-based learning module 340 according to theembodiment of the present disclosure creates a video summary byselecting a corresponding key frame based on the predicted importancescore, evaluates the quality of the created video summary, and performspolicy gradient learning for the attention-based video summarizationnetwork.

The policy gradient algorithm-based learning module 340 according to theembodiment of the present disclosure performs parameterized policygradient learning by computing a set of normalized importance weightsfor each episode by using an objective function of an explorationstrategy for exploring an under-appreciated reward (UREX) method and asoftmax function for approximation of the objective function.

The video summarization network module 320 according to the embodimentof the present disclosure calculates regularization and reconstructionloss for controlling the probability to select a key frame by using theimportance score of the selected key frame. The video summary creationmodule 350 according to the embodiment of the present disclosure createsa video summary based on the calculated regularization andreconstruction loss.

FIG. 4 is a flowchart 400 for explaining an unsupervised video summarymethod with an efficient key frame selection reward function accordingto an embodiment of the present disclosure.

The proposed unsupervised video summary method with an efficient keyframe selection reward function includes: the step 410 in which aframe-level image feature extraction module extracts frame-level visualfeatures from an input video; the step 420 in which an attention-basedvideo summarization network module computes an attention weight andrepresents an importance score as a frame tracking probability forselecting a key frame by using the attention weight; the step 430 inwhich an evaluation module obtains a temporal consistency rewardfunction and a representativeness reward function so as to select thekey frame, based on a visual similarity distance and temporal distancebetween key frames, and trains an attention-based video summarizationnetwork to predict an importance score for selecting a key frame of avideo summary by using the temporal consistency reward function and therepresentativeness reward function; the step 440 in which a policygradient algorithm-based learning module creates a video summary byselecting a corresponding key frame based on the predicted importancescore, evaluates the quality of the created video summary, and performspolicy gradient learning for the attention-based video summarizationnetwork; the step 420 in which the video summarization network modulecalculates regularization and reconstruction loss for controlling theprobability to select a key frame by using the importance score of theselected key frame; and the step 450 in which a video summary creationmodule creates a video summary based on the calculated regularizationand reconstruction loss.

In step 410, the frame-level image feature extraction module extractsframe-level visual features from an input video.

In step 420, the attention-based video summarization network modulecomputes an attention weight and represents an importance score as aframe tracking probability for selecting a key frame by using theattention weight.

According to the embodiment of the present disclosure, an encodernetwork, a decoder network, and an attention layer between the encodernetwork and the decoder network reduce parameters and calculationsthrough dilated RNN and extract temporal dependency.

The encoder network captures visual similarities with local and globalcontext between key frames, and the attention layer computes anattention weight by using both the output of the encoder network and thelast hidden state of the decoder network.

The attention weight is normalized to a probability score of each keyframe by a softmax function, and a context vector is obtained bymultiplying the output of the encoder network by the attention weight.

The decoder network is trained by connecting the context vector and theprevious output of an initialized decoder network for the input of thedecoder network, to obtain an importance score by using learning resultsof the decoder network and the encoder network.

In step 430, the evaluation module obtains a temporal consistency rewardfunction and a representativeness reward function so as to select thekey frame, based on a visual similarity distance and temporal distancebetween key frames, and trains the attention-based video summarizationnetwork to predict an importance score for selecting a key frame of avideo summary by using the temporal consistency reward function and therepresentativeness reward function.

The evaluation module is trained to calculate the similarity between theselected key frame and all key frames of the video is calculated byusing features extracted using the representative reward function andpredict an importance score for selecting a key frame of a video summarythrough the representative reward function.

The evaluation module is trained to repeat the process of finding theclosest neighbor among the selected key frames with respect to all thekey frames, in order to select a representative shot-level key frameusing the temporal consistency reward function.

In step 440, the policy gradient algorithm-based learning module createsa video summary by selecting a corresponding key frame based on thepredicted importance score, evaluates the quality of the created videosummary, and performs policy gradient learning for the attention-basedvideo summarization network.

According to the embodiment of the present disclosure, the policygradient algorithm-based learning module performs parameterized policygradient learning by computing a set of normalized importance weightsfor each episode by using an objective function of an explorationstrategy for exploring an under-appreciated reward (UREX) method and asoftmax function for approximation of the objective function.

Back in step 420, the video summarization network module calculatesregularization and reconstruction loss for controlling the probabilityto select a key frame by using the importance score of the selected keyframe. In step 450, the video summary creation module creates a videosummary based on the calculated regularization and reconstruction loss.The proposed unsupervised video summarization method with an efficientkey frame selection reward function will be described in more detailwith reference to FIGS. 5 to 7 .

FIG. 5 is a view 500 illustrating a configuration of an attention-basedvideo summarization network for computing an attention weight accordingto an embodiment of the present disclosure.

First, extract visual feature

{x_(t)}_(t = 1)^(N)

are extracted from an input video using GoogleNet [19], which is a deepconvolutional neural network trained with an ImageNet dataset. Featureextraction is needed to capture visual features of frame images with alow-dimensional feature vector. The extracted features help efficientlycalculate the visual differences among the frames in the video.

The present disclosure proposes an attention-based video summarizationnetwork for predicting a key frame-level importance score, as shown inFIG. 5 . The attention-based video summarization network includes anencoder network 510, a decoder network 520, and an attention layer 530between the two networks. This network improves an attention autoencoderby replacing an LSTM network with a dilated RNN by the attentionautoencoder [4] proposed with respect to a SUM-GANAAE method andreplacing a GRU (gate recurrent unit) network with a decoder. Thedilated RNN is implemented together with a dilated skip connection inorder to improve computational efficiency with less parameters. Inparticular, the network extracts temporal dependency by stacking dilatedrecurrent layers and exponentially increasing dilations across all thelayers. Also, GRU cells are used as the layers of the dilated RNN. Theencoder network captures visual similarities with local and globalcontext between key frames, and the attention layer computes anattention weight by using both the output of the encoder network and thelast hidden state of the decoder network. To compute the attentionweight, the attention mechanism proposed in [20] is used. In particular,as explained in Equation (1), a content-based score function is used tocalculate the attention weight by using the output of the encodernetwork Eout and the last hidden state of the decoder network ht-1. Fort=1, the output of the decoder network h is set to 0. Wα is attentionweight matrix which are parameters that can be learned:

$\begin{matrix}{Score\left( {E_{out},h_{t - 1}} \right) = E_{out}^{T}W_{a}h_{t - 1}} & \text{­­­Equation (1)}\end{matrix}$

Next, the attention weight is normalized to a probability score of eachkey frame by a softmax function, and a context vector C is obtained bymultiplying the output of the encoder network by the attention weight.

The context vector and the previous output of the decoder networkinitialized to 0 are connected for the input of the decoder networkD_(in). Also, the decoder network is trained.

Then, the next output D_(out) is obtained and connected to E_(out) toreuse rich information on which the encoder network is trained, therebyimproving performance for a long-sequence video. Also, the dimension ofa feature size to be transmitted to the decoder network in the next stept+1, along with a linear function, is reduced. Lastly, by using a fullyconnected layer and a sigmoid function, the dimension of the output isreduced, and an output is produced using the importance score

S = {s_(t)}_(t = 1)^(N).

That is, the importance score is a frame tracking probability (0. to 1.)for selecting a key frame as a video summary.

FIG. 6 is a view 600 for explaining a representativeness reward functionaccording to an embodiment of the present disclosure.

For efficient video summarization according to the embodiment of thepresent disclosure, a diversity reward function in the conventionaltechnology [6] is adopted, and two new rewards are proposed forefficient key frame selection. The proposed reward functions are atemporal consistency reward function and a representativeness rewardfunction, which allow for a visual similarity distance and temporaldistance between key frames.

The diversity reward function R_(div) Equation (2) calculatesdifferences between key frames selected by a frame select action, alongwith extracted features. Through this reward function, the network istrained to predict an importance score for selecting diverse frames askey frames of a summary. Also, the summary consisting of such key framesallows easy grasping of what the video is about. The temporal distanceis limited to 20 for the calculation of the differences between selectedkey frames, in order to keep the storyline of the video and reduce thecomputational complexity. Without this limitation, even if a flashbackscene or a similar scene is far from the selected key frames, thesescenes can be ignored when diverse frames are selected.

Let the indices of the selected key frames be

𝒥 = {i_(k)|a_(i_(k)) = 1, k = 1, 2, ..., |𝒥|},

then the diversity reward is:

$\begin{matrix}{R_{div} = \frac{1}{J\left( {J - 1} \right)}{\sum\limits_{t \in J}{\sum\limits_{\begin{array}{l}{t^{\prime} \in J} \\{t \neq t^{\prime}}\end{array}}\left( {1 - \frac{x_{t}^{T}x_{t^{\prime}}}{\left\| x_{t} \right\|_{2}\left\| x_{t^{\prime}} \right\|_{2}}} \right)}}} & \text{­­­Equation (2)}\end{matrix}$

The representativeness reward function R_(rep) Equation (3) calculatesthe similarity between the selected key frame and all key frames of thevideo. Through this reward function, the network is trained to predictan importance score for selecting a key frame of a summary representingthe video. The summary consisting of key frames allows easy grasping ofthe subject matter of the video. The present disclosure proposes a newtechnique of applying an importance score S as the representativenessreward function, as in D × st represented in Equation (3) in order tocreate a good video summary and improve performance. To train thenetwork, the representativeness reward function needs to be increased,and the distance D Equation (4) in the reward function needs to beminimized.

$\begin{matrix}{R_{rep} = exp\left( {- \frac{1}{N}{\sum_{t = 1}^{N}{D_{t} \times s_{t}}}} \right)} & \text{­­­Equation (3)}\end{matrix}$

$\begin{matrix}{D_{t} = \underset{t^{\prime} \in J}{min}\left\| {x_{t} - x_{t^{\prime}}} \right\|_{2}} & \text{­­­Equation (4)}\end{matrix}$

Referring to FIG. 6 , A is an example in which the distance between aselected key frame and all key frames is long, and B is an example inwhich the distance between a selected key frame and all key frames isshort. If the importance score S of a key frame is high, the key framemay be selected for a summary. On the other hand, if the importancescore of a key frame is low, the probability to select the key frame fora summary is low. Both A and B are the average of D × S, and A is higherthan B as shown in FIG. 6 . Thus, the reward for A is lower than thereward for B based on the proposed reward function.

There are three cases that explain the effects of a proposed trick usingimportance scores.

1. If the distance D between key frames is short, the reward functionreturns a low reward even if the importance score is changed.

2. If the distance D between key frames is long and the importance scoreS of a key frame is high, the reward function returns a high reward.Also, the network is trained with the low reward, and then the networkpredicts a score of low importance for the key frame in order to preventselection.

3. If the distance D between key frames is long and the importance scoreS of a key frame is low, the reward function returns an intermediatereward. However, most of the key frames will not be selected as asummary because the importance score is low.

FIG. 7 is a view for explaining a temporal consistency reward functionaccording to an embodiment of the present disclosure.

A temporal consistency reward function Rcon Equation (6) is proposed toselect a representative shot-level key frame efficiently and uniformly.

As explained in FIG. 7 , the similarity between a selected key framesuch as

V_(i)^(3^(′))

of

V_(i)^(summary)

and other frames of

V_(i)^(all)

is calculated, and then the most similar frame

V_(i)³

other than itself is selected.

To explain temporal consistency rewards, such representative key framesas key frames included in B of FIG. 7 are defined as having a similarscene around a neighboring frame. On the other hand, the key framesincluded in A of FIG. 7 have a similar scene that is temporarily farfrom a selected key frame. Depending on the storyline intended by theproducer, a similar scene to a key frame included in A may appear aftera while or may be seen only once. However, one problem with the keyframes included A is that a summary made up of these hinders the user’sunderstanding of the video content. Thus, these key frames are to beremoved from the summary. Another advantage of removing these key framesis to prevent excessive selection from one side. That is, key frames maybe uniformly selected. In the present disclosure, the process of findingthe closest neighbor among the key frames

selected from all key frames

{x_(t)}_(t = 1)^(N)

until the number

|𝒥|

of all key frames is repeated. In the present disclosure, the temporalconsistency of a summary may be learned by minimizing the distancebetween

and

. To minimize the distance, the reward function is calculated asfollows:

$\begin{matrix}{j_{k} = \left( {\underset{k \in {|J|}}{argmin}\left\| {x_{i_{k}} - x} \right\|} \right)} & \text{­­­Equation (5)}\end{matrix}$

$\begin{matrix}{R_{can} = \frac{1}{\log\left( {\sum_{k = 1}^{J}{\left( {i_{k} - j_{k}} \right)^{2}/|J|}} \right)}} & \text{­­­Equation (6)}\end{matrix}$

To normalize rewards, the distance is divided by

|𝒥|,

and log probabilities are used.

To select a key frame as a summary, the Bernoulli distribution is usedwhich is a discrete probability for converting the importance score Sinto a frame tracking action A = {α_(t)|α_(t) ∈ {0,1}, t = 1,..,N}. Ifthe frame-selection action of a frame is equal to 1, this key frame isselected as a summary. Since the Bernoulli distribution randomlygenerates variants of frame-selection action, it promotes exploringvarious summaries of the video.

$\begin{matrix}{\left. A \right.\sim Bernoulli\left( {a_{t};s_{t}} \right) = \left\{ \begin{matrix}{s_{t},} & {for\text{­­­Equation (7)}a_{t} = 1} \\{1 - s_{t},} & {for\mspace{6mu} a_{t} = 0}\end{matrix} \right)} & \end{matrix}$

Also, the quality of a generated summary is evaluated by the sum ofrewards. Through this reward, the attention-based video summarizationnetwork is trained with a parameterized policy using the policy gradientmethod. The policy gradient method is one of the reinforcement learningmethods for exploring an action strategy to obtain a more efficientsummary using a gradient descent method. To avoid a lack of actionstrategy, the objective function of an exploration strategy forexploring an under-appreciated reward (UREX) method is used [21]. If thelog probability log π_(θ)(a_(t) | h_(t)) of the action π_(θ)(a_(t) |h_(t)), under the policy, underestimates its reward

$r\left( {a_{t}\left| h_{t} \right)} \right) = \frac{R_{rep} + R_{div}}{2} + R_{can},$

then action will be explored more by the exploration strategy.

To compute the objective function of

UREX𝒪_(UREX),

the log probability of the action and the reward in

𝒥

episodes is maintained.

𝒪_(UREX)

is an expected value of the reward R(a_(t) | h_(t)) which is a sum ofRAML (Reward Augmented Maximum Likelihood) objective function. In thepresent disclosure, to approximate the RAML objective function, the setof normalized importance weights for each episode j is computed usingthe softmax function:

$\begin{matrix}{O_{\mspace{6mu} UREX}\left( {\theta;\tau} \right) = \mathbb{E}_{h\sim p{(h_{t})}}\left\{ {\sum\limits_{a \in A}{R\left( {a_{t}\left| h_{t} \right)} \right)}} \right\}} & \text{­­­Equation (8)}\end{matrix}$

In the present disclosure, the base line is used, which is an importantmethod for policy gradient, to reduce variance and to improvecomputational efficiency. The baseline is calculated as the movingaverage of rewards experienced so far. To improve diversity using themoving average of various videos, the baseline is calculated by addingthe baseline for each video b 1 and the baseline for all videos b₂ asfollows. Lastly, L_(rwd) is maximized as a cost for training thenetwork.

$\begin{matrix}{B = 0.7 \times b_{1} + 0.3 \times b_{2}} & \text{­­­Equation (9)}\end{matrix}$

$\begin{matrix}{L_{rwd} = O_{\mspace{6mu} UREX}\left( {\theta;\tau} \right) - B} & \text{­­­Equation (10)}\end{matrix}$

In the present disclosure, the regularization term L_(reg) which isproposed in the conventional technology [6] to control the probabilityto select key frames using importance scores. If most of the importancescores is close to 1 or close to 0, the probability to select wrong keyframes as a summary is increased. Accordingly, L_(reg) is used toapproximate the importance score to 0.5 during training. The importancescore is multiplied by 0.01 to avoid it from converging fast to 0.5.

$\begin{matrix}{L_{reg} = 0.01 \times \left( {\frac{1}{N} \times {\sum\limits_{1}^{N}{s_{t} - 0.5}}} \right)^{2}} & \text{­­­Equation (11)}\end{matrix}$

After computing all of the loss function, the loss for video summaryL_(summary) is finally calculated, and backpropagation is done:

$\begin{matrix}{L_{summary} = L_{reg} - L_{rwd}} & \text{­­­Equation (12)}\end{matrix}$

Algorithm 1 is about the training procedure of the unsupervised videosummarization method with the policy gradient method.

Algorithm 1 Training the Network 1: Input: x_(r) frame-level features ofthe video 2: Output: proposed network’s parameters (θ) 3: 4: for thenumber of iterations do 5: S ← Network(x_(t)) % Predict the importancescore S 6: A ← Bernoulli Distribution(S) % Action A from the score S 7:% Calculate the reward functions and the loss using A and S 8:$\left\{ \theta \right\}\overset{+}{\leftarrow} - \nabla(L_{reg} - L_{rwd})\%\text{Minimization}$9: % Update the network using the policy gradient method: 10: end for

According to the embodiment of the present disclosure, the shot-levelimportance scores are calculated by averaging the frame-level importancescores within the shot, to test the attention-based video summarizationnetwork. For a performance comparison with other methods, the shot-levelimportance scores of a video are needed. To detect shots, the KernelTemporal Segmentation (KTS) method is used which detects change points,such as shot boundaries [22]. To generate the video summary, key shotsover the top 15% of the video length sorted by the score are selected.This step is the same concept as the ‘0-1 Knapsack problem formaximizing the importance of a summary video, as described in theconventional technology [6].

The aforementioned apparatus may be implemented in the form of ahardware component, a software component, and/or a combination of ahardware component and a software component. For example, the system andcomponents described in the embodiments may be implemented using one ormore general-purpose computers or special-purpose computers, such as aprocessor, a controller, an arithmetic logic unit (ALU), a digitalsignal processor, a microcomputer, a field programmable gate array(FPGA), a programmable logic unit (PLU), a microprocessor, or any otherdevice capable of executing or responding to an instruction. A processormay run an operating system (OS) and one or more software applicationsexecuted on the OS. Furthermore, the processor may access, store,manipulate, process, and generate data in response to the execution ofsoftware. For convenience of understanding, one processing device hasbeen illustrated as being used, but a person having ordinary skill inthe art may understand that the processor may include a plurality ofprocessing elements and/or a plurality of types of processing elements.For example, the processor may include a plurality of processors or asingle processor and a single controller. Furthermore, a differentprocessing configuration, such as a parallel processor, is alsopossible.

Software may include a computer program, code, an instruction, or acombination of one or more of these and may configure a processor sothat it operates as desired or may instruct the processor independentlyor collectively. The software and/or data may be embodied in a machine,component, physical device, virtual equipment, computer storage mediumor device of any type in order to be interpreted by the processor or toprovide an instruction or data to the processor. The software may bedistributed to computer systems connected over a network and may bestored or executed in a distributed manner. The software and data may bestored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form ofa program instruction executable by various computer means and stored ina computer-readable recording medium. The computer-readable recordingmedium may independently or collectively include program instructions,data files, data structures, and the like. The program instructionsrecorded in the medium may be specifically designed and configured forthe embodiments, or may be known to and used by those of ordinary skillin the computer software art. Examples of the medium include a magneticmedium such as a hard disk, a floppy disk and a magnetic tape, anoptical recording medium such as CD-ROM and DVD, a magneto-opticalmedium such as a floptical disk, and a hardware device such as ROM, RAM,and flash memory, which is specifically configured to store and executeprogram instructions. Examples of the program instructions may includemachine-language code, such as code written by a compiler, andhigh-level language code executable by a computer using an interpreter.

As described above, although the embodiments have been described inconnection with the limited embodiments and the drawings, those skilledin the art may modify and change the embodiments in various ways fromthe description. For example, the relevant results may be achieved evenwhen the described technologies are performed in a different order thanthe described methods, and/or even when the described components such assystems, structures, devices, and circuits are coupled or combined in adifferent form than the described methods or are replaced or substitutedby other components or equivalents.

Therefore, other implementations, other embodiments, and equivalents tothe claims are also within the scope of the following claims.

References

Ejaz, N.; Mehmood, 1.; Baik, S.W. Efficient visual attention basedframework for extracting key frames from videos. J. Image Commun. 2013,28, 34-44.

Gygli, M.; Grabner, H.; Riemenschneider, H.; Gool, L.V. Creatingsummaries from user videos. In Proceedings of the European Conference onComputer Vision (ECCV)); Springer, 2015, pp. 505-520.

Yoon, U,N.; Hong, M.D.; Jo, G.S. Interp-SLTM: Unsupervised VideoSummarization with Piecewise Linear Interpolation. Sensors 2021. vol.21, no. 13, 4562

Apostolidis, E.; Adamantidou, E.; Metsai, A.; Mezaris, V.; Patras, I.Unsupervised Video Summarization via Attention-Driven AdversarialLearning. In International Conference on Multimedia Modeling (MMM);Springer: Daejeon, Korea, 5-8 Jan. 2020, pp. 492-504.

Jung, Y.J.; Cho, D.Y.; Kim, D.H.; Woo, S.H.; Kweon I.S. Discriminativefeature learning for unsupervised video summarization. AAAI Conferenceon Artificial Intelligence, Honolulu, Hawaii, USA, 27 January - 1 Feb.2019, pp. 8537-8544.

Zhou, K.; Qiao, Y.; Xiang, T. Deep Reinforcement Learning forUnsupervised Video Summarization with Diversity-RepresentativenessReward. AAAI Conf. Artif. Intell. 2018, 32, 7582-7589.

Song, Y.; Vallmitjana, J.; Stent, A.; Jaimes, A. Tvsum: Summarizing webvideos using titles. In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition, Boston, MA, USA, 7-12 Jun. 2015, pp.5179-5187

Feng, L.; Li, Z.; Kuang, Z.; Zhang, W. Extractive Video Summarizer withMemory Augmented Neural Networks. MM, Seoul, Republic of Korea, 22-26Oct. 2018, pp. 976-983

Zhang, K.; Chao, W.L.; Sha, F.; Grauman, K. Video Summarization withLong Short-term Memory. In Proceedings of the European Conference onComputer Vision (ECCV); Springer: Amsterdam, Netherlands, 2016; pp.766-782.

Zhang, Y.; Kampffmeyer, M.; Zhao, X.; Tan, M. DTR-GAN: Dilated TemporalRelational Adversarial Network for Video Summarization. In Proceedingsof the ACM Turing Celebration Conference (ACM TURC), Shanghai, China, 18May 2018.

Ji. Z.; Xiong. K.; Pang. Y.; Li. X. Video Summarization withAttention-Based Encoder-Decoder Networks. IEEE Transactions on circuitsand systems for video technology, June 2020, vol. 30, no. 6. pp.1709-1717.

Mahasseni, B.; Lam, M.; Todorovic, S. Unsupervised Video Summarizationwith Adversarial LSTM Networks. In Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition, Honolulu, Hawaii, USA, 22-25Jul. 2017, pp. 202-211.

Yuan, L.; Tay, F.E.; Li, P.; Zhou, L.; Feng, F. Cycle-SUM:Cycle-consistent Adversarial LSTM Networks for Unsupervised VideoSummarization. In Proceedings of the Thirty-Third AAAI Conference onArtificial Intelligence, Honolulu, Hawaii, USA, 27 Jan - 1 Feb. 2019,Volume 33, pp. 9143-9150.

Kaufman, D.; Levi, G.; Hassner, T.; Wolf, L. Temporal Tessellation: AUnified Approach for Video Analysis. In Proceedings of the IEEEInternational Conference on Computer Vision (ICCV), 2017, pp. 94-104.

Rochan, M.; Ye, L.; Wang, Y. Video Summarization Using FullyConvolutional Sequence Networks. In Proceedings of the EuropeanConference on Computer Vision (ECCV); Springer: Munich, Germany, 2018,pp. 347-363.

Silver, D.; Lever, G.; Heess, N.; Degris, T.; Wierstra, D.; Riedmiller,M. Deterministic Policy Gradient Algorithms. In Proceedings of the 31stInternational Conference on International Conference on Machine Learning(ICML), Beijing, China, 21-26 Jun. 2014, pp. 387-395.

Yu, Y. Towards Sample Efficient Reinforcement Learning. In Proceedingsof the Twenty-Seventh International Joint Conference on ArtificialIntelligence (IJCAI), Stockholm, Sweden, 13-19 Jul. 2018, pp. 5739-5743.

Lehnert, L.; Laroche, R.; Seijen, H.V. On Value Function Representationof Long Horizon Problems. In Proceedings of the Thirty-Second AAAIConference on Artificial Intelligence, New Orleans, Louisiana, USA, 2-7Feb. 2018, pp. 3457-3465.

Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.;Erhan, D.; Vanhoucke, B.; Rabinovich, A. Going deeper with convolutions.In Proceedings of the IEEE Conference on Computer Vision and PatternRecognition (CVPR), Boston, MA, USA, 7-12 Jun. 2015, pp. 1-9.

Luong. T.; Pham. H.; Manning C.D.; Effective Approaches toAttention-based Neural Machine Translation. Proceedings of the 2015Conference on Empirical Methods in Natural Language Processing (EMNLP),Lisbon, Portugal, 17-21 Sep. 2015, pp. 1412-1421.

Nachum, O.; Norouzi, M.; Schuurmans, D. Improving Policy Gradient byExploring Under-Appreciated Rewards. arXiv 2016, arXiv:1611.09321.

Potapov, D.; Douze, M.; Harchaoui, Z.; Schmid, C.; Category-specificvideo summarization. European Conference on Computer Vision (ECCV),Zurich, Switzerland, September 2014, pp.540-555.

Rochan, M.; Wang, Y. Video Summarization by Learning from Unpaired Data.In Proceedings of the IEEE Conference on Computer Vision and PatternRecognition (CVPR) 2019, pp. 7902-7911.

1. An attention-based video summarization method comprising: extractingframe-level visual features from an input video; computing an attentionweight and representing an importance score as a frame trackingprobability for selecting a key frame by using the attention weight;obtaining a temporal consistency reward function and arepresentativeness reward function so as to select the key frame, basedon a visual similarity distance and temporal distance between keyframes, and training an attention-based video summarization network topredict an importance score for selecting a key frame of a video summaryby using the temporal consistency reward function and therepresentativeness reward function; creating a video summary byselecting a corresponding key frame based on the predicted importancescore, evaluating the quality of the created video summary, andperforming policy gradient learning for the attention-based videosummarization network; calculating regularization and reconstructionloss for controlling the probability to select a key frame by using theimportance score of the selected key frame; and creating a video summarybased on the calculated regularization and reconstruction loss.
 2. Theattention-based video summarization method of claim 1, wherein, thecomputing of the attention weight and representing the importance scoreas a frame tracking probability for selecting a key frame by using theattention weight is performed using an encoder network, a decodernetwork, and an attention layer between the encoder network and thedecoder network that reduce parameters and calculations through dilatedRNN and extract temporal dependency, wherein the encoder networkcaptures visual similarities with local and global context between keyframes, wherein the attention layer computes an attention weight byusing both the output of the encoder network and the last hidden stateof the decoder network, wherein the attention weight is normalized to aprobability score of each key frame by a softmax function, wherein acontext vector is obtained by multiplying the output of the encodernetwork by the attention weight, and wherein the decoder network istrained by connecting the context vector and the previous output of aninitialized decoder network for the input of the decoder network, toobtain an importance score by using learning results of the decodernetwork and the encoder network.
 3. The attention-based videosummarization method of claim 1, wherein the obtaining of the temporalconsistency reward function and the representativeness reward functionso as to select the key frame, based on a visual similarity distance andtemporal distance between key frames, and trains an attention-basedvideo summarization network to predict an importance score for selectinga key frame of a video summary by using the temporal consistency rewardfunction and the representativeness reward function further includes:calculating the similarity between the selected key frame and all keyframes of the video is calculated by using features extracted using therepresentative reward function and predicting an importance score forselecting a key frame of a video summary through the representativereward function, and repeating the process of finding the closestneighbor among the selected key frames with respect to all the keyframes, in order to select a representative shot-level key frame usingthe temporal consistency reward function.
 4. The attention-based videosummarization method of claim 1, wherein the creating of the videosummary by selecting the corresponding key frame based on the predictedimportance score, evaluating the quality of the created video summary,and performing policy gradient learning for the attention-based videosummarization network, further includes: performing parameterized policygradient learning by computing a set of normalized importance weightsfor each episode by using an objective function of an explorationstrategy for exploring an under-appreciated reward (UREX) method and asoftmax function for approximation of the objective function.
 5. Anattention-based video summarization apparatus comprising: a processor;and a memory coupled to the processor, the memory containinginstructions, that when executed by the processor: extracts frame-levelvisual features from an input video; computes an attention weightthrough an attention-based video summarization network and represents animportance score as a frame tracking probability for selecting a keyframe by using the attention weight; obtains a temporal consistencyreward function and a representativeness reward function so as to selectthe key frame, based on a visual similarity distance and temporaldistance between key frames, and trains an attention-based videosummarization network to predict an importance score for selecting a keyframe of a video summary by using the temporal consistency rewardfunction and the representativeness reward function; creates a videosummary by selecting a corresponding key frame based on the predictedimportance score, evaluates the quality of the created video summary,and performs policy gradient learning for the attention-based videosummarization network, wherein the video summarization network modulecalculates regularization and reconstruction loss for controlling theprobability to select a key frame by using the importance score of theselected key frame; and creates a video summary based on the calculatedregularization and reconstruction loss.
 6. The attention-based videosummarization apparatus of claim 5, further comprising: an encodernetwork, a decoder network, and an attention layer between the encodernetwork and the decoder network that reduce parameters and calculationsthrough dilated RNN and extract temporal dependency, wherein the encodernetwork captures visual similarities with local and global contextbetween key frames, wherein the attention layer computes an attentionweight by using both the output of the encoder network and the lasthidden state of the decoder network, wherein the attention weight isnormalized to a probability score of each key frame by a softmaxfunction, wherein a context vector is obtained by multiplying the outputof the encoder network by the attention weight, and wherein the decodernetwork is trained by connecting the context vector and the previousoutput of an initialized decoder network for the input of the decodernetwork, to obtain an importance score by using learning results of thedecoder network and the encoder network.
 7. The attention-based videosummarization apparatus of claim 5, wherein the similarity between theselected key frame and all key frames of the video is calculated byusing features extracted using the representative reward function and animportance score for selecting a key frame of a video summary ispredicted through the representative reward function, and is trained torepeat the process of finding the closest neighbor among the selectedkey frames with respect to all the key frames, in order to select arepresentative shot-level key frame using the temporal consistencyreward function.
 8. The attention-based video summarization apparatus ofclaim 5, wherein the instructions, when executed by the processor,further performs parameterized policy gradient learning by computing aset of normalized importance weights for each episode by using anobjective function of an exploration strategy for exploring anunder-appreciated reward (UREX) method and a softmax function forapproximation of the objective function.